The present invention relates in general to a method and system for implementing data transfers utilizing Direct Memory Access (DMA). More particularly, the present invention relates to implementing User-level Direct Memory Access (UDMA) data transfers.
Attention is called to the following publications: Matthias A. Blumrich, et al., "Virtual-Memory-Mapped Network Interfaces", IEEE Micro, Vol. 15, No. 1, pages 21-28 (February 1995); Matthias A. Blumrich, et al., "A virtual memory mapped network interface for the SHRIMP multicomputer", Proc. of 21st Int'l Symp. on Computer Architecture, pages 142-153 (April 1994); John L. Hennessy and David A. Patterson, "Computer Architecture: A Quantitative Approach", Morgan Kaufmann, (1996); A. S. Tanenbaum, "Modern Operating Systems", Prentice Hall, (1992); M. Morris Mano, "Computer System Architecture", Prentice Hall, (1982).
Definitions are included to facilitate the discussion of the present invention and the prior art. Other definitions are interspersed throughout the Detailed Description of the Invention as necessary. The definitions are as follows:
Atomicity--Indivisibility of operations. Two operations comprise an atomic sequence if they happen in direct succession, with no intervening operations. PA1 Backplane--Also known as motherboard. The printed circuit board on which is mounted the processor, memory and input/output (I/O) logic of a personal computer (PC) or similar small system. PA1 Context Switch--A change in which process is running on the CPU. The running process is stopped and the entire state of its computation is saved. Then another process is started from the point at which it has been stopped. PA1 Central Processing Unit (CPU)--The processor or microprocessor which executes all programs, including the operating system. The processor is time-shared. PA1 Device--A hardware component or sub-system. Memory is one type of device. PA1 Engine--A hardware state machine. PA1 Interrupt--A hardware event which forces a context switch to special operating system code known as interrupt handler code. PA1 Memory Management Unit (MMU)--Translates a processes' virtual addresses to real physical addresses based on the processes' page tables. PA1 Polling--Periodically checking status, such as that of a device. PA1 Process--A computation having its own virtual address space. PA1 Protection--Restricting access to a device so that multiple processes or users may safely share the device. PA1 Pinning--Prevents moving a page of data out of memory by virtual memory processing. PA1 System Call--A function call to the operating system. This causes a context switch to the operating system. PA1 System Level--The operating system. System level code normally uses the privileged mode of the CPU operation which allows the full instruction set of the processor to be used. PA1 User-level--Applications. User-level code normally does not use the privileged mode of the CPU operation, so a restricted set of instructions is allowed. PA1 1. Saving the context PA1 2. Checking permission in order to enforce protection PA1 3. Address translation (virtual to physical) PA1 4. Checking to see if the DMA engine is available PA1 5. Pinning memory pages PA1 6. Setting up DMA addresses PA1 7. Starting the DMA PA1 8. Unpinning the pages when the DMA transfer is complete PA1 9. Restoring the context
DMA is a common technique for routing data directly between memory and an I/O device without requiring intervention by the CPU to control each datum transferred. DMA was first implemented on the IBM SAGE computer in 1955 and has always been a common approach in I/O interface controller designs for data transfer between main memory and I/O devices. For example, Local Area Network (LAN) adapters and Small Computer Systems Interface (SCSI) devices typically have DMA capability.
In a multiple process computer system, a DMA transaction can be initiated only through the operating system kernel in order to achieve protection, memory buffer management, and related address translation. The overhead of this kernel-initiated DMA transaction is often hundreds, possibly thousands of CPU instructions. As a result, DMA has been found beneficial only for infrequent operations which transfer large amounts of data, restricting its usefulness.
An increasing need for a data transfer using DMA techniques which requires minimal operating system overhead has been recognized by the computer industry. This is because many computer I/O device interface designs require low overhead DMA initiations. An example of this is the network interface of a workstation cluster, a multicomputer or multiprocessor, all of which require low-latency message passing. Another example is the graphics display interfaces which move data from main memory to graphics frame buffers.
A problem occurs utilizing traditional DMA controllers or devices since the high overhead of these devices requires coarse grained transfers of large data blocks in order to achieve the available raw DMA channel bandwidths. This is particularly true for high-bandwidth devices such as network interfaces and High Performance Parallel Interface (HIPPI) devices.
For example, the overhead of sending a piece of data over a 100 Mbyte/sec HIPPI channel on a computer such as the Paragon multicomputer is more than 350 microseconds. With a data block size of 1 Kbyte, the transfer rate achieved is only 2.7 MByte/sec, which is less than 2% of the raw hardware bandwidth. Achieving a transfer rate of 80 MByte/sec requires the data block size to be larger than 64 KBytes. The high overhead is the dominating factor which limits the utilization of DMA controllers for fine grained data transfers.
Another serious consideration with a kernel-initiated DMA transaction, is that protection is provided by the operating system because it is the only program allowed access to the configuration registers of the DMA controller. Processes must perform system calls to access the DMA controller. The system call is necessary primarily to verify the user's permission to access the device, and to ensure mutual exclusion among processes sharing the device. A system call requires a context switch to the operating system and eventually a context switch back to the application. The operating system call entails:
The time required to perform the context switches and to run the operating system is considered the overhead of accessing the DMA controller. User processes must pay the overhead of a system call to initiate a DMA operation.
Another disadvantage with a kernel-initiated or traditional DMA transaction is the cost of pinning and unpinning affected pages or alternatively copying pages into special pre-pinned I/O buffers. The physical memory pages used for DMA data transfer must be pinned to prevent the virtual memory system from paging them out while DMA data transfers are in progress. Since the cost of pinning memory pages is high, most of the systems implemented today reserve a certain number of pinned physical memory pages for each DMA controller as I/O buffers. This method may require copying data between memory in user address space and the reserved, pinned DMA memory buffers.
Another drawback of a kernel-initiated DMA transaction is the requirement of virtual-to-physical memory address translation. Because the DMA controller uses physical memory addresses, the virtual memory addresses of the user programs must be translated to physical addresses before being loaded into the DMA address registers. Virtual-to-physical address translation has a high overhead because it is performed by the operating system kernel.